RETINAL OCT IMAGE ANALYSIS USING DEEP LEARNING

Optical coherence tomography (OCT) is a noninvasive imaging modality which uses low-coherence light waves to take cross-sectional images of optical scattering media. OCT has been widely used in diagnosing retinal and neural diseases by imaging the human retina. The thicknesses of retinal layers are important biomarkers for neurological diseases like multiple sclerosis (MS). The peripapillary retinal nerve fiber layer (pRNFL) and ganglion cell plus inner plexiform layer (GCIP) thickness can be used to assess the global disease progression of MS patients. Automated OCT image analysis tools are critical for quantitatively monitoring disease progression and exploring biomarkers. With the development of more powerful computational resources, deep learning based methods have achieved much better performance in accuracy, speed, and algorithm flexibility for many image analysis tasks. However, without task-specific modifications, these emerging deep learning methods are not satisfactory if directly applied to tasks like retinal layer segmentation. In this thesis, we present a set of novel deep learning based methods for OCT image analysis. Specifically, we focus on automated retinal layer segmentation from macular OCT images. The first problem we address is that existing deep learning methods do not incorporate explicit anatomical rules and cannot guarantee the layer segmentation hierarchy~(pixels of the upper layers should have no overlap or gap with pixels of layers beneath it). To solve this, we developed an efficient fully convolutional network to generate structured layer surfaces with correct topology that is also able to perform retinal lesion~(cysts or edema) segmentation. The second problem we addressed is that the segmentation uncertainty reduces the sensitivity of detecting mild retinal changes in MS patients over time. To solve this, we developed a longitudinal deep learning pipeline that considers both inter-slice and longitudinal segmentation priors to achieve a more consistent segmentation for monitoring patient-specific retinal changes. The third problem we addressed is that the performance of the deep learning models will degrade when test data is generated from different scanners~(domain shift). We address this problem by developing a novel test-time domain adaptation method. Different from existing solutions, our model can dynamically adapt to each test subject during inference without time-consuming retraining. Our deep networks achieved state-of-the-art segmentation accuracy, speed, and flexibility compared to the existing methods

3D Automatic Segmentation Method for Retinal Optical Coherence Tomography Volume Data Using Boundary Surface Enhancement

With the introduction of spectral-domain optical coherence tomography (SDOCT), much larger image datasets are routinely acquired compared to what was possible using the previous generation of time-domain OCT. Thus, there is a critical need for the development of 3D segmentation methods for processing these data. We present here a novel 3D automatic segmentation method for retinal OCT volume data. Briefly, to segment a boundary surface, two OCT volume datasets are obtained by using a 3D smoothing filter and a 3D differential filter. Their linear combination is then calculated to generate new volume data with an enhanced boundary surface, where pixel intensity, boundary position information, and intensity changes on both sides of the boundary surface are used simultaneously. Next, preliminary discrete boundary points are detected from the A-Scans of the volume data. Finally, surface smoothness constraints and a dynamic threshold are applied to obtain a smoothed boundary surface by correcting a small number of error points. Our method can extract retinal layer boundary surfaces sequentially with a decreasing search region of volume data. We performed automatic segmentation on eight human OCT volume datasets acquired from a commercial Spectralis OCT system, where each volume of data consisted of 97 OCT images with a resolution of 496 512; experimental results show that this method can accurately segment seven layer boundary surfaces in normal as well as some abnormal eyes.Comment: 27 pages, 19 figure

Automated 3D Segmentation of Kidneys and Tumors in MICCAI KiTS 2023 Challenge

Kidney and Kidney Tumor Segmentation Challenge (KiTS) 2023 offers a platform for researchers to compare their solutions to segmentation from 3D CT. In this work, we describe our submission to the challenge using automated segmentation of Auto3DSeg available in MONAI. Our solution achieves the average dice of 0.835 and surface dice of 0.723, which ranks first and wins the KiTS 2023 challenge.Comment: MICCAI 2023, KITS 2023 challenge 1st plac

Aorta Segmentation from 3D CT in MICCAI SEG.A. 2023 Challenge

Aorta provides the main blood supply of the body. Screening of aorta with imaging helps for early aortic disease detection and monitoring. In this work, we describe our solution to the Segmentation of the Aorta (SEG.A.231) from 3D CT challenge. We use automated segmentation method Auto3DSeg available in MONAI. Our solution achieves an average Dice score of 0.920 and 95th percentile of the Hausdorff Distance (HD95) of 6.013, which ranks first and wins the SEG.A. 2023 challenge.Comment: MICCAI 2023, SEG.A. 2023 challenge 1st plac

Probing Ground-state Single-electron Self-exchange Across A Molecule-metal Interface

We have probed single-molecule redox reaction dynamics of hemin (chloride) adsorbed on Ag nanoparticle surfaces by single-molecule surface-enhanced Raman spectroscopy (SMSERS) combined with spectroelectrochemistry. Redox reaction at the molecule/Ag interface is identified and probed by the prominent fluctuations of the Raman frequency of a specific vibrational mode, nu(4), which is a typical marker of the redox state of the iron center in a hemin molecule. On the basis of the autocorrelation and cross-correlation analysis of the single-molecule Raman spectral trajectories and the control measurements of single-molecule spectroelectochemistry and electrochemical STM, we suggest that the single-molecule redox reaction dynamics at the hemin Ag interface is primarily driven by thermal fluctuations. The spontaneous fluctuation dynamics of the single-molecule redox reaction is measured under no external electric potential across the molecule metal interfaces, which provides a novel and unique approach to characterize the interfacial electron transfer at the molecule metal interfaces. Our demonstrated approaches are powerful for obtaining molecular coupling and dynamics involved in interfacial electron transfer processes. The new information obtained is critical for a further understanding, design, and manipulation of the charge transfer processes at the molecule metal interface or metal-molecule-metal junctions, which are fundamental elements in single-molecule electronics, catalysis, and solar energy conversion

Automated segmentation of intracranial hemorrhages from 3D CT

Intracranial hemorrhage segmentation challenge (INSTANCE 2022) offers a platform for researchers to compare their solutions to segmentation of hemorrhage stroke regions from 3D CTs. In this work, we describe our solution to INSTANCE 2022. We use a 2D segmentation network, SegResNet from MONAI, operating slice-wise without resampling. The final submission is an ensemble of 18 models. Our solution (team name NVAUTO) achieves the top place in terms of Dice metric (0.721), and overall rank 2. It is implemented with Auto3DSeg.Comment: INSTANCE22 challenge report, MICCAI2022. arXiv admin note: substantial text overlap with arXiv:2209.0954

HI-TOM: A Benchmark for Evaluating Higher-Order Theory of Mind Reasoning in Large Language Models

Theory of Mind (ToM) is the ability to reason about one's own and others' mental states. ToM plays a critical role in the development of intelligence, language understanding, and cognitive processes. While previous work has primarily focused on first and second-order ToM, we explore higher-order ToM, which involves recursive reasoning on others' beliefs. We introduce HI-TOM, a Higher Order Theory of Mind benchmark. Our experimental evaluation using various Large Language Models (LLMs) indicates a decline in performance on higher-order ToM tasks, demonstrating the limitations of current LLMs. We conduct a thorough analysis of different failure cases of LLMs, and share our thoughts on the implications of our findings on the future of NLP.Comment: Accepted at Findings of EMNLP 202

Automated head and neck tumor segmentation from 3D PET/CT

Head and neck tumor segmentation challenge (HECKTOR) 2022 offers a platform for researchers to compare their solutions to segmentation of tumors and lymph nodes from 3D CT and PET images. In this work, we describe our solution to HECKTOR 2022 segmentation task. We re-sample all images to a common resolution, crop around head and neck region, and train SegResNet semantic segmentation network from MONAI. We use 5-fold cross validation to select best model checkpoints. The final submission is an ensemble of 15 models from 3 runs. Our solution (team name NVAUTO) achieves the 1st place on the HECKTOR22 challenge leaderboard with an aggregated dice score of 0.78802.Comment: HECKTOR22 segmentation challenge. MICCAI 2022. arXiv admin note: text overlap with arXiv:2209.0954